Skip to content

spike: Serve robots.txt#21623

Draft
sp94sap wants to merge 1 commit into
developfrom
spike/ai-seo/CXSPA-13220
Draft

spike: Serve robots.txt#21623
sp94sap wants to merge 1 commit into
developfrom
spike/ai-seo/CXSPA-13220

Conversation

@sp94sap

@sp94sap sp94sap commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Problem

Requests to /robots.txt fall through to the Angular SSR engine, which returns server-rendered HTML with Content-Type: text/html. This is a hard failure for all crawlers — traditional and AI alike.

Solution

Add an explicit Express route that intercepts /robots.txt before the Angular SSR catch-all and serves a plain text response.

Changes

  • core-libs/setup/ssr/robots-txt/ — new module with createRobotsTxtHandler(options?), RobotsTxtOptions interface, and DEFAULT_ROBOTS_TXT_CONTENT
  • core-libs/setup/ssr/public_api.ts — exports new module
  • core-libs/schematics/src/add-ssr/files/server.ts — schematic template updated so new projects get the route wired automatically
  • projects/storefrontapp/src/server.ts — demo app updated
  • projects/ssr-tests/src/ssr-robots-txt.spec.ts — SSR E2E tests

Usage

  Default behavior — no config needed:                                                                                                                                                                                                  
  const robotsTxtHandler = createRobotsTxtHandler();        
  if (robotsTxtHandler) {                                                                                                                                                                                                               
    server.get('/robots.txt', robotsTxtHandler);                                                                                                                                                                                        
  }

Custom content override:

  createRobotsTxtHandler({                                                                                                                                                                                                              
    content: `                                              
  User-agent: *                                                                                                                                                                                                                         
  Allow: /                                                                                                                                                                                                                              
  Disallow: /checkout/                                                                                                                                                                                                
  Sitemap: https://my-storefront.example.com/sitemap.xml                                                                                                                                                                                
  `                                                                                                                                                                                                                                     
  })

Disable entirely:
createRobotsTxtHandler({ enabled: false }) // returns null, route not registered

Out of scope

Content Signals, per-bot config toggles, and CSR static asset support are deferred to follow-up tasks.

Test plan

  • nx run setup:test — unit tests pass
  • npm run test:ssr (requires built SSR app + backend) — E2E test ssr-robots-txt.spec.ts passes
  • Manual: curl -I http://localhost:4000/robots.txt returns Content-Type: text/plain

@sp94sap sp94sap requested a review from a team as a code owner June 16, 2026 10:59
@github-actions github-actions Bot marked this pull request as draft June 16, 2026 11:00
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

@sp94sap sp94sap changed the title feat: Serve /robots.txt from Express before Angular SSR engine (CXSPA-13220) feat: Serve robots.txt Jun 16, 2026
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

@sp94sap sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 1d30fef to 6219cd6 Compare June 16, 2026 11:46
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

@sp94sap sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 6219cd6 to 5dae578 Compare June 16, 2026 12:33
@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

spike(CXSPA-13220) serve /robots.txt from Express SSR before Angular …
b438e19 
…engine

  Previously, requests to /robots.txt fell through to the Angular SSR engine
  and returned HTML with Content-Type: text/html, breaking all crawlers.
  - Adds createRobotsTxtHandler() to @spartacus/setup/ssr — returns an
    Express RequestHandler that serves plain text with Cache-Control headers,
    or null when disabled (enabled: false)
  - Ships DEFAULT_ROBOTS_TXT_CONTENT with rules for traditional crawlers and
    major AI crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot,
    Google-Extended, Claude-SearchBot, ChatGPT-User)
  - Accepts RobotsTxtOptions.content for full merchant override
  - Route is registered before static assets and the Angular SSR catch-all
  - Updates the add-ssr schematic template so new projects get it wired automatically
  - Adds unit tests and SSR E2E test

  Content Signals and per-bot config toggles are out of scope for this spike.
  CSR-only deployments must serve robots.txt via static hosting / CDN.
@sp94sap sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 5dae578 to b438e19 Compare June 17, 2026 08:12
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

@sp94sap sp94sap changed the title feat: Serve robots.txt spike: Serve robots.txt Jun 18, 2026
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

  • docs: Changes to documentation only
  • feat: New feature work
  • fix: Bug fixes
  • perf: Code improvements for performance
  • refactor: Code changes that are not bug fixes or features
  • style: Code style changes (e.g., whitespace, formatting)
  • test: Adding or updating tests
  • chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

rmch91
rmch91 approved these changes Jun 23, 2026
View reviewed changes

@rmch91 rmch91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rmch91 rmch91 rmch91 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sp94sap @rmch91